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An Evaluation of One- and Three-Parameter 
Logistic Tailored Testing Procedures 
for use with Small Item Pools 



Tailored testing has shown considerable promise as an 
alternative to conventional paper-and-pencil testing, but 
before it can be implemented on a widescale basis, a number 
of issues must be addressed. Tailored testing procedures 
involve a number of complex components, and there are often 
a number of alternatives which may bs chosen for each. 
Although there has been considerable research conducted in 
this area, it is still unclear which of tl.e many alternative 
components should be us*t=a in any particular application. 
For instance, one important component of tailorecTtesting is 
the item response theory(IRT) model upon which the procedure 
is to be based. There are numerous IRT models, several of 
which have been proposed for use in tailored testing. The 
purpose of this study was to compare tailored testing 
procedures based on two of the most popular IRT models, the 
one-parameter lostistic (1PL) and three-parameter logistic 
(3PL) models, to determine whether one of the two models is 
preferable to the other in a tailored achievement testing 
setting. The tailored testing procedures based on the 1PL 
and 3PL models were compared on the basis of the ability 
estimates which were yielded by the procedures. Before 
reporting the results of the study, it may be helpful to 
review previous research comparing tailored testing 
procedures based on these two models. 

Compariso ns of 1PL and 3PL Tailored Testing Procedures 

Several studies have been conducted to compare the use of 
the 1PL and 3PL models for tailored testing. One such 
study, reported by Koch and Reckase (1978), was a direct 
comparison of 1PL and 3PL tailored testing procedures in an 
application to vocabulary measurement. Both procedures 
employed maximum likelihood ability estimation techniques, 
and in both procedures items were selected to maximize the 
information function at the current ability estimate. The 
results of this study indicated that both models could be 
successfully applied to vocabulary ability measurement. The 
3PL procedure had a slightly higher reliability (a cross 
between test-retest and equivalent forms reliabilities) than 
the 1PL procedure (r * .77 for the 3PL procedure, r = ,61. 
for the 1PL procedure). However, the 3PL procedure failed 
to converge to ability estimates in nearly one third of the 
cases, while nonconvergence was not a serious problem with 
the 1PL procedure. 
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In a second study / reported by Koch and Reckase (1979) / 
1PL and 3PL tailored testing procedures were applied to a 
multidimensional achievement test. Results of this study 
indicated very poor performance for both procedures, 
primarily due to small sample sizes, poor linking 
procedures, and poor selection of the stepsize and initial 
ability estimates for the maximum likelihood estimation 
procedure. 

A study reported by McKinley and Reckase (1980) attempted 
to correct the problems encountered in the Koch and Reckase 
studies. Close attention was paid to appropriate item 
parameter linking and selection of the operating 
characteristics of the procedures. The results of this 
study indicated that both models could be quite successfully 
applied to tailored achievement testing if correctly 
implemented. Both 1PL and 3PL reliabilities were higher 
than the reliability of a classroom test over the same 
material. The 3PL procedure yielded better fit to the data 
than the 1PL procedure / and it also yielded higher test 
information than the 1PL procedure. This study concluded 
that for tailored achievement testing the 3PL model was the 
model of choice. However, the test used in this study was 
highly multidimensional. It is unclear how generalizable 
the results are to less multidimensional achievement test. 

Urry (1970, 1977) also concluded that the 3PL model was 
the model of choice. Through a series of simulation studies 
Urry found that tailored testing becomes less effective when 
a model with an insufficient number of parameters is used. 
He concluded that construct valdity decreases as a function 
of the degree of degeneracy of the model, and the 1PL model 
was particularly inappropriate for use with multiple-choice 
items because it did not portray multiple-choice response 
data with fidelity (Urry, 1977). 

This review of previous research indicates that if 
careful attention is paid to all components of the tailored 
testing procedure, both 1PL and 3PL tailored testing can be 
successful. The 3PL model tends to yield higher 
reliabilities and test information than the 1PL procedure, 
but is more prone to complications such as nonconvergence . 
It is also indicated that the 3PL model yields better fit to 
multidimensional data*. Thus, the results of these studies 
tend to favor the 3PL model. Of course, these results were 
obtained using relatively large item pools. It is unclear 
from these studies what results would be. obtained using 
smaller item pools. The purpose of this study was to 
compare the 1PL and 3PL models in a tailored achievement 
testing applicati on for which a relatively small item pool 

7 

o 

ERJC 



-3- 



is available. 

Method 

Models 

The two models selected for this study were the one- 
parameter logistic (1PL) and the three-parameter logistic 
(3PL) models. The 1PL model is given by 

exp((G :j -b i )x i:j ) 



1+exptG^-b.) 

where 0,. is the ability parameter for examinee j/ b^ is the 
difficulty parameter for item i, x^ is the observed score 
(0 or 1) on item i for examinee j, and P(*jj) is the 
probability of response x^j to item i by examinee j. The 
3PL model is given by 

ex R (Da.<e.-b.)) 

Ptx.^l) = c.+<l~c.) — 

l+exptDa.^-L.)) 

where c^ is the pseudo-guessing parameter for item i, a^ is 

the discrimination parameter for item i, where ( 0 j ) is the 

probability of a correct response to item i by examinee j, 
and the remaining terms are as previously defined. 

Estimation P rograms 

For both the 1PL and the 3PL models parameters were 
estimated using the LOGIST program (Wingersky, Barton/ and 
Lord/ 1982). For the 1PL model the pseudo-guessing 
parameter was held fixed at 0,0. The discrimination 
parameter was held fixed at a value computed by the LOGIST 
program. To check the 1PL estimates obtained from LOGIST/ 
they were compared to parameter estimates obtained for the 
same data using the MAX program (Wright and Panchapakesan/ 
1969), which was designed for use with Uie 1PL model. Since 
the results obtained from the two programs were almost 
identical, LOGIST was used throughout the study. The LOGIST 
program was used for both models in order to avoid problems 
due to different parameter estimate scales. For both models 
the scales were based on the ability estimate distributions. 

Tailored Testing Procedures 

Tailored testing procedures have three main components; 
an item selection routine/ an ability estimation technique/ 
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and a stopping rule. In this study both the 1PL and 3PL 
procedures selected items to maximize the value of the 
information function (Birnbaum, 1968) at the most recent 
ability estimate. The information for each item at the 
examinee's current ability estimate was computed, and the 
item with the greatest information at that ability estimate 
was administered / with the provision that the information 
had to be greater than 0.226 for the 1PL procedure and 0.450 
for the 3PL procedure. These values wesre selected on the 
basis of several trial runs. They were selected so as to 
yield approximately equal average test lengths for the two 
models. For both procedures 20 items was the maximum test 
length allowed. 

Prior to testing initial estimates of ability were 
assigned to set the starting points in the item pool. The 
initial ability estimates for this. study were set to be 
0.221 for the 1PL procedure and 0.420 for the 3PL procedure. 
These values represent difficulty values near the medians of 
the item pool difficulty parameter distributions. The first 
item was then selected to maximize information at the 
initial ability estimate. The response of the examinee to 
that item was then simulated in the following manner. For 
the first part of the study, response data came from a fixed 
length, non-tailored test comprised of all the items in the 
pool. These items had been administered in paper and pencil 
form to all of the examinees used in this study. An 
examinee's response to an item in the tailored tests was the 
actual response of the examinee to the item on the paper and 
pencil test. For the second part of the study, simulated 
response data were generated for each examinee for each item 
in the pool. These data were generated according to the 3PL 
model using the 3PL item parameter estimates obtained for 
the real response data and examinee abilities selected t 
random from a standard normal distribution. These responses 
were used regardless of whether a 1PL or 3PL based tailored 
test were used. 

Once the response by an examinee to an item had been 
obtained, a new estimate of ability was computed by adding a 
fixed stepsise to the old ability estimate if the response 
were correct, and by subtracting a fixed stepsiEe if the 
response were incorrect. This fixed stepsise procedure was 
used until a maximum likelihood ability estimate could be 
obtained (i.e., when both correct and incorrect responses 
were obtained). The stepsise used was 0.300 for both 
procedures. Each new item was selected to maximise the 
information at the new ability estimate, with the 
restriction that no item could be used more than once. 
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Two stopping rules were used for the tailored testing 
procedures. The tests were terminated when there were no 
items left in the item pool with information at the current 
ability estimate greater than the minimum specified above / 
or when 20 items had been administered. 

Design 

This study employed a two-staye design — one involving the 
use of real data, and one involving simulated data. In the 
first stage of the study, response data were obtained for a 
larqe sample on a relatively short paper and pencil test. 
Part of the large sample was then used to calibrate the 
items on the test using both the 1PL and 3PL models. Using 
the resulting item parameter estimates / 1PL and 3PL tailored 
tests were simulated for the examinees not included in the 
calibration sample. The responses by the examinees to the 
items in the tailored tests were the same responses they 
made to the items when taking the paper and pencil test. 

In the second stage of the study, the item parameter 
estimates obtained from the 3PL calibration of the paper and 
pencil test were used as true parameters / along with the 
true abilities selected at random from the standard normal 
distribution to generate simulated response data to fit the 
3PL model. Data were generated for a large sample for all 
the items from the paper and pencil test. The procedure 
used for the real data part of the study was then repeated 
using these simulated data* 

Data 

For the real data part of the study / response data for 
the 40 item Mathematics Usage subtest of the ACT Assessment 
(The American College Testing Program / 1982) were obtained 
for 3000 cases from the October, 1982 administration of the 
ACT Assessment (Form 23B) , For the second stage of the 
study, data were simulated for 40 items and 3000 cases. For 
both stages, then, rather small item pools were used. 

Analyses 

The analyses performed in this study consisted primarily 
of computing and comparing correlations. For both the real 
and the simulation data, the 1PL and 3PL tailored test 
aMlity estimates were compared by computing the correlation 
between them. For the simulation data the two sets of 
ability estimates obtained from the tailored tests were also 
compared to the true abilities used to generate the data. 
Again, the comparisons were performed using correlations. 
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Results 



Real Data Analyses 

Item Pool Calibration The first analysis performed on the 
real data was the calibration of the items for use as a 
tailored testing item pool'. The calibration of the items / 
which was based on response data for the first 2000 
examinees / was performed three different ways. The first 
two calibrations were performed for the 1PL model using the 
LOGIST and MAX programs while the third was performed for 
the 3 PL model using LOGIST. The MAX and LOGIST 1PL item 
difficulty parameter estimates had a correlation of 0.999, 
as did the ability estimates obtained from the two program. 
This comparison was performed in order to determine whether 
the LOGIST program could be used for both models throughout 
the study. These findings indicated that it could / thus 
simplifying the problem of placing the estimates from the 
two models on the same scale. 

The item parameter estimate distributions obtained for 
the two models using LOGIST are shown in Figure 1. These 
distributions are summarized by the statistics shown in 
Table 1. As can be seen, most of the 3PL discrimination 
parameter estimates were .60 or higher, so most of the items 
were of fairly high quality. From the 3PL difficulty 
parameter estimate distribution, however, it: can be seen 
that the items are appropriate only for a limited range of 
ability, since most of the item difficulty estimates fall in 
the range from -1*0 to 1.75. Most of guessing parameter 
estimates are .3 or less, with only two items having 
guessing parameter estimates greater than .3. From these 
data it would appear that these items actually form a fairly 
high quality item pool for tailored testing, except for the 
limitation on the range of difficulty. 

For the 1PL model, the LOGIST program assigned to all 
items a discrimination value of 0.561. The pseudo-guessing 
parameter was, of course, 0.0. The 1PL difficulty parameter 
estimate distribution is somewhat different from the 3PL 
difficulty distribution although the two sets of estimates 
had a correlation of .88, with the biggest difference being 
a shift downward of the bulk of the estimates for the 1PL 
model. Most of the difficulty parameter estimates fall 
within the same range as for the 3PL model, but there 
appears to be a shift toward the negative end of that range. 
Still, for that range the items form an item pool of fairly 
high quality. 



Figure 1 



The 1PL and 3PL Item Parameter Estimate 
Frequency Distributions for the Real Data 
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Table 1 

Descriptive Statistics of Item Parameter 
Estimates for the Real Data 



1PL 



3PL 



Statistic 



b 



a 



b 



c 



Mean 

Median 

S,D, 

Skewness 
Kurtosis 
Low Value 
High Value 



0,03 
0.22 
0.91 
0,24 
0.19 
2,07 
2.04 



0,98 
0.90 
0.34 
0.40 
0.04 
0.31 
1.81 



0.46 
0.41 
1.10 
0.20 
0.99 
2.12 
3.15 



0.17 
0, 16 
0,08 
1.14 
1.19 
0.08 
0.41 



Figure 2 shows the test information function for the item 
pool based on the 1PL item parameter estimates, while Figure 
3 shows the test information function based on the 3PL 
estimates. As can be seen from Figure 3, the 3PL curve is 
negatively skewed, and is centered around 1.0, thus yielding 
more information for the positive end of the ability scale* 
The 1PL curve, on the other hand, is not skewed, and is 
centered around 0.2. It would appear from this, then, that 
the 1PL item parameter estimates are appropriate for a wider 
range for ability than the 3PL estimates are. Of course, 
the ability scales are not exactly comparable because they 
are based on different item parameters < 

Ability Estimates For those examinees not included in the 
calibration sample, four different estimates of ability were 
computed. For each examinee a 1PL and 3PL ability estimate 
was obtained from simulated tailored test. In addition, 
ability estimates for each examinee for both models were 
obtained from LOGIST using the item parameter estimates and 
the examinee responses from the 40 item paper and pencil 
test. This made possible not only a comparison of the two 
tailored testing procedures, but also a comparison of the 
tailored testing procedures with the paper and pencil tests. 

Table 2 summarizes the distributions of the ability 
estimates obtained for both models from the tailored tests 
and from the paper and pencil tests. Table 3 shows the 
intercorrelation matrix for these four sets of ability 
estimates. As can be seen from these data, the two sets of 
tailored test ability estimates were similar, with a 
correlation of 0.77. However, there were some differences 
in the two distributions. 
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TIig Test Information Function for the 1PL 
Item Parameter Estimates for the Real Data 
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Figure 3 

The Test Information Function for the 3PL 
Item Parameter Estimates for the Real Data 
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For instance, the skewness value of -0.97 for 3PL ability 
estimate distribution was significantly different from zero 
(with a sample size of 1000, the standard error for the 
skewness coefficient is 0.08), while the 1PL ability 
estimate distribution ->as not significantly skewed. Also, 
the kurtosis value of 1.96 for the 3PL ability estimate 
distribution was significant (standard error = 0.16)* while 
the kurtosis value of the 1PL ability estimate distribution 
was not significant. 

Table 2 

Descriptive Statistics of Ability Parameter 
Estimates for the Real Data 



Tailored Tests Paper and Pencil Tests 

Statistic 

1PL 3 PL 1PL 3PL 



Mean 


0. 


15 


0. 


01 


0 


21 


0. 


11 


Median 


0. 


14 


0. 


23 


0 


16 


0 


25 


S.D. 


1. 


36 


1. 


40 


1 


13 


1 


18 


Skewness 


0 


10 


-0. 


97 


0 


74 


-0 


35 


Kurtosis 


0 


21 


1. 


96 


3 


.48 


4 


39 


Low Value 


-3 


65 


-4. 


00 


-2 


.92 


-4 


00 


High Value 


6 


22 


6. 


42 


4 


00 


4 


00 


Mean Test Length 


12 


84 


12 


16 


40 


00 


40 


00 


S.D. of Test Length 


4 


51 


4 


73 


0 


.00 


0 


00 



Note. For the LOGIST calibrations arbitrary minimums and 
maximums of -4.00 and 4.00 were set on the ability 
estimates. The same limits were placed on the tailored 
tests except in those cases where all items were answered 
correctly or all were answered incorrectly. 



Table 3 

Intercorrelation Matrix for Ability Parameter 
Estimates for the Real Data 



Ability 




Tailored 


Tests 


Paper and 


Pencil Tests 


Estimate 




1PL 


3 PL 


1PL 


3 PL 


Tailored 


1PL 


1.00 


0.77 


0.89 


0.87 




3 PL 




1,00 


0.81 


0.86 


Paper/Pencil 


1PL 






1,00 


0.95 




3PL 








1.00 
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The 1FL and 3PL ability estimates from the paper and 
pencil test had a correlation of 0.95. Both distributions 
were leptokurtic (kurtosis =3.48 for the 1PL estimates, 
4.39 for the 3PL estimates), and the two distributions had 
similar means nnd standard deviations. The only real 
deference between these two distributions was that the 3PL 
distribution v/as significantly negatively skewed (skewness = 
-0.35), while the 1PL distribution was significantly 
positively skewed (skewness = 0.74). 

The two sets of tailored test ability estimates were 
fairly similar to the paper and pencil test ability 
estimates. The two sets of 1FL estimates had a correlation 
of 0,89, and the two sets of 3PL estimates had a correlation 
of 0.86. A comparison of these two correlations via 
Fisher's r to z transformation yields a z = 2.20, ]3 < .05, 
indicating that the 1PL correlation was significantly higher 
than the 3PL correlation. Interestingly, the 3PL tailored 
test ability estimates had a correlation with the 3PL paper 
and pencil test estimates which was not significantly 
different from the correlation between the 1PL tailored test 
ability estimates and the 3PL paper and pencil test ability 
estimates (r = 0.86 for the 3PL estimates, 0.87 for the 1PL 
estimates). The 1PL tailored test ability estimates did 
have a significantly higher correlation with the 1PL paper 
and pencil test estimates than did the 3PL tailored test 
ability estimates (r = 0.89 versus r = 0.81). 



A verage Test Length The average test length for the 1PL 
tailored tests was 12.8 items, while the average 3PL 
tailored test was 12.2 items long. This difference is of 
little or no practical importance, except as an indication 
that the attempt to produce tests of equal length for the 
two models was successful. Of some importance is the 
finding that the 1PL tailored tests required approximately 
one half of the CPU time required by the 3PL procedures. Of 
course, if this difference had no signicant impact on 
response time, then it also is of no practical significance. 

Nonconvergence For the 1PL procedure there was no 
nonconvergence , For the 3PL procedure, however, there was a 
4.9% nonconvergence rate, Examinees for v hom there was 
nonconvergence were assigned an ability estimate of 4.0 or 
"4.0. Of those case© where there was nonconvergence, 96% 
were at the low end of ability. This is consistent with the 
finding that the 3PL test information curve was negatively 
skewed and shifted toward the positive end. Nonconvergence 
here means that the tailored testing procedure was not able 
to compute an ability estimate for an examinee. This could 
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happen because the examinee answered all the items 
correctly, or all the items incorrectly. It could also 
happen if the examinee's ability estimate drifted out of the 
range for which there were appropriate items before both an 
incorrect and a correct response were obtained. In such a 
case, the tesc would be terminated at 20 items, or when both 
a correct and an incorrect answer were obtained. 

Simulation Data Analyses 

I tern Pool Calibration The first step in the simulation 
data stage of this study was the generation of data to fit 
the 3PL model. The true item parameters used for these data 
were the 3PL item parameter estimates obtained for the real 
data used in the first part of the study. Data were 
generated for 3000 cases, using true ability parameters 
randomly selected from the standard normal distribution. 
Once these data were generated, the items were calibrated 
for both the 1PL and 3PL models using the first 2000 cases. 
The distributions of the obtained item parameter estimates 
are shown in Figure 4. These distributions are summarized 
bj the statistics shown in Table 4. 

Table 4 

Descriptive Statistics of Item Parameter 
Estimates for the Simulation Data 



1PL 



3PL 



Statistic 



Mean 


O.nO 


1,04 


0.41 


0.17 


Median 


0.16 


0.96 


0.30 


0,14 


S.D. 


0.90 


0.34 


1.12 


0.08 


Skewness 


-0.31 


0. 13 


0.09 


0,94 


Kurtosis 


0, 38 


0.11 


1.86 


1.10 


Low Value 


-2.20 


0.28 


-2.27 


0,06 


High Value 


2.00 


1.77 


3,77 


0,40 



With feu exceptions, these distributions are very much 
like the distributions of the item parameter estimates 
obtained for the real data. The only real differences were 
in the skewness of the 3PL model a-values, which went from 
slightly positively skewed to not significantly skewed, and 
the kurtosis of the b-values for the 3PL model, which had an 
increased kurtosis for the simulation data. 



Figure 4 



TIig iPL and 3PL Item Parameter Estimate 
Frequency Distributions for the Simulation Data 
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One other important difference that was found was that for 
the 1PL calibration the items were assigned in a-value of 
0.60. Since this was higher than tha value for the real 
data (0.56), it was expected that the test information curve 
for the 1PL model would be higher for the simulation data 
than for the real data. It was unclear what effect this 
would have on the simulated 1PL tailored tests, except that 
it would probably increase the average test length; 

Table 5 shov/s the intercorrelation matrix for the true 
and estimated i tem parameters for the simulation data . As 
can be seen, the 3PL estimates wer^ quite similar to the 
true parameters. The correlations of the true and estimated 
3PL item parameters were 0.89 for the a-values, 0.99 for the 
b-values, and 0.92 for the c-values. The correlation of the 
1PL b-values with the true b-values was 0.88, and the 
correlation of the 1PL and 3PL b-value estimates was 0.88. 

Table 5 

Intecorrelation Matrix for the True and Estimated 
Item Parameters for the Simulation Data 



Item True 1PL Estimates 3PL Estimates 



Parameter a 


b 




c 




b 




a 


b 


c 


True a 1.00 


0.25 


0. 


10 


0 


45 


0 


89 


0.21 


-0.09 


b 


1.00 


0. 


40 


0 


88 


0 


27 


0.99 


0.29 


c 




1. 


00 


0 


11 


0 


19 


0.34 


0.92 


1PL b 








1 


00 


0 


41 


0.88 


-0.04 


3PL a 












1 


00 


0.23 


0.08 


b 
















1.00 


0.26 


c 


















1,00 



Figures 5 and 6 show the test information curves for the 
1PL and 3PL item parameter estimates / respectively. As was 
the case with the real data, the 3PL information curve is 
shifted toward the positive end of the ability scale. it is 
centered around .8, The JLPL curve / on the other hand, is 
centered around 0.0. The 1PL pool once again appears to be 
appropriate for a wider range of ability than the 3PL pool 
is, especially at the lower end of the ability scale. As 
was predicted from the item calibration results, the 1PL 
test information curve was higher for the simulation data 
than for the real data. An unexpected result was that the 
3PL test information curve was also higher for the 
simulation data than for the real data. This was probably 
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Figure 5 



The Test Information Function for the 1PL 
Item Parameter Estimates for the Simulation Data 

o 




Figure 6 

The Test Information Function for the 3FL 
Item Parameter Estimates for the Simulation Data 
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a result of the fact that the simulation data ere generated 
to fit the 3PL model. 

Ability Estimates Four sets of ability estimates were 
once again computed for the 1000 examinees not included in 
the calibration sample. For each simulated examinee 1PL and 
3PL ability estimates were obtained from the simulated 
tailored tests as well as from LOGIST runs on the simulated 
40 item fixed length test using the item parameter estimates 
from the calibration of the simulation data, Thus / all the 
comparisons made with the real data results could be made 
with the simulation data results. Because these were 
simulation data and the true ability parameters were known, 
the "ability estimates obtained for these data could also be 
compared to the true abilities. 

The statistics shown in Table 6 summarize the true 
ability parameter distribution, as well as all of the 
ability estimate distributions obtained using the simulation 
data. Table 7 shows the intercorrelation matrix for the 
true and estimated abilities for the simulation data. The 
patterns appearing in these data are much like those found 
for the real data. For these data the correlations are all 
higher than for the real data, however, with the exception 
of the correlation between the 1PL and 3PL (simulated) paper 
and pencil test ability estimates, which was lower for the 
simulation data (0.928 versus 0.946 for the real data). The 
1PL tailored test ability estimates had a correlation of 
0.931 with the 1PL simulated paper and pencil test 
estimates, which was significantly higher than the 
■correlation of 0,826 obtained between the 3PL tailored test 
estimates and the 1PL paper and pencil test estimates ( z ~ 
10.954, g < .01). The 1PL and 3PL tailored test estimates 
had correlations of 0.920 and 0.854, respectively, with the 
3PL paper and pencil test estimates. The difference between 
these two correlations is significant ( z = 7.113, g < .01), 
indicating that the 1PL correlation was significantly 
greater than the 3PL correlation. 

The inclusion of the true ability parameters in the 
analyses of the simulation data resulted in a very 
interesting finding. While the 1PL and 3PL paper and pencil 
test estimates had correlations with the true parameters 
that were not significantly different (0.894 for the 3PL 
estimates, C.900 for the 1PL estimates), the correlation of 
the 1PL tailored test ability estimates with the true 
abilities was significantly higher than the correlation of 
the 3PL tailored tests ability estimates with the true 
abilities (r = .883 for the 1PL estimates, 0.816 for the 3PL 
estimates; z = 5.452, g < .01). This was rather surprising 
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since the simulation data were generated to fit the 3PL 
model. Just as surprising was the finding that the 1PL 
tailored test ability estimates had a correlation with the 
true abilities that was not significantly less than the 
correlations between the true abilities and the paper and 
pencil test estimates / despite the fact that the maximum 
length of the tailored tests was only half the length of the 
paper and pencil tests. 

Table 6 

Descriptive Statistics of True and Estimated Abilities 
for the Simulation Data 



Tailored Tests Paper and Pencil Tests 



Statistic 


True 


1PL 


3PL 


1PL 


3PL 


Mean 


-0.01 


-0.08 


-0.25 


0.02 


-0.09 


Median 


0.00 


-0.07 


0.00 


-0.10 


0.03 


S.D. 


1.04 


1.30 


1.48 


1.11 


1.22 


Skewness 


-0.01 


0.32 


-0.58 


1.11 


-0.24 


Kurtosis 


0.14 


0.86 


1.52 


4.27 


4.04 


Low Value 


-3.82 


-3.61 


-5.58 


-2.47 


-4.00 


High Value 


3.74 


6.22 


6.42 


4.00 


4.00 


Mean Test 












Length 




17.90 


13.51 


40.00 


40.00 


S.D. of Te 


St 










Length 




4.05 


5.77 


0.00 


0.00 



Mot e. For the L0GIST calibrations arbitrary minimums and 
maximums of -4.00 and 4.00, respectively, were set on the 
ability estimates. The same limits were placed on the 
tailored tests except in those cases where all items were 
answered correctly or all were answered incorrectly. 

Table 7 

Intercorrelation Matrix for True and Estimated Abilities 

for the Simulation Data 



Ability Tailored Tests Paper and Pencil Tests 

True 

Estimate 1PL 3PL 1PL 3 PL 



True 1.00 0.88 

Tailored 1PL 1.00 
3PL 

P&P 1PL 
3PL 



0.82 0.90 0.89 

0.81 0.93 0.92 

1.00 0.83 0.85 

1.00 0.93 
1.00 
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Average Test Le ng th The average test length of the 3PL 
tailored tests for the simulation data was 13*5 items. The 
average 1PL tailored test was 17.9 items long. Both of 
these averages were greater for the simulation data than for 
the real data as was predicted from the results of the test 
information curve analyses. The average, 3PL test increased 
by 1.3 while the r aver age 1PL test increased by 5.1. The 
increased length of the 1PL tests for the simulation data 
could at least partially explain why the 1PL tailored test 
estimates had higher correlations with the true abilities 
and the paper and pencil test estimates than the 3PL 
tailored test estimates did. Despite the longer average 
length of the 1PL tailored test, it should be pointed out 
that the 3PL procedure required half again as much CPU time 
as the 1PL procedure, 

Nonconvergence The 1PL procedure had a .3% nonconvergence 
rate, while the 3PL procedure had a 5.9% nonconvergence 
rate. For the 1PL procedure all of the nonconvergence cases 
(three of them) were at the positive end of the ability 
scale. For the 3PL procedure 90% of the nonconvergence 
cases were at the low end of the ability scale. As was the 
case with the real data, examinees for whom there was 
nonconvergence were assigned an ability estimate of 4.0 or 
-4.0. 

Discussion 

In recent years a number of studies reported in the 
literature have addressed the issue of whether the 1PL model 
or the 3PL model should be used in various tailored testing 
applications. In a tailored achievement testing 
application, the application of interest here, the research 
has tended to favor the 3PL model. Because of the 
inconclusiveness of these studies for applications involving 
small item pools, and because the 3PL model tends to be more 
expensive to use, this study was conducted to determine, for 
a specific application, whether there is sufficient 
advantage to using the 3PL model to warrant the extra 
expense. The results of this study will now be discussed, 
and afterwards some conclusions regarding which model should 
be used for this application will be presented. First, 
however, a discussion of the specific application which is 
of interest in this study will be presented* 

The Application 

The specific application of interest here has several 
characteristics which require special consideration. The 
type of application of concern is an achievement testing 
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appiication. Achievement testing must be considered in a 
different light than ability testing because it is learning 
rather than ability that is being measured. While ability 
tests generally have learning components, they are 
constructed to measure a single trait, and as such are 
usually reasonably unidimensional . Achievement tests, on 
the other hand, are not specifically directed at a single 
trait . Moreover * achievement tests of cen are designed to 
measure learning in a number of content areas. Therefore, 
achievement tests typically are not unidimensional, and are 
often highly multidimensional. The multidimensionality of 
achievement tests causes problems for IRT, since most IRT 
models assume unidimensionaiity . 

One way to deal with the dimensionality problem when 
measuring achievement via IRT is treat the different content 
areas separately. Individual content areas typically are 
not unidimensional, but they at least afford a closer 
approximation to unidimensionalty than do mult i~con tent area 
tests. Treating content areas separately presents a new 
problem for tailored testing. A single content area of a 
test may not include very many items. Tailored testing 
procedures work best when the item pool has a relatively 
large number of items, with difficulties spread uniformly 
over the ability range (Urry, 1977). Building an item pool 
to meet those specifications, but using only items from a 
single contant area might be difficult, and certainly would 
be time-consuming . It seems likely, then, that at least in 
the early stages of a tailored achievement testing program 
that treats content areas separately the item pools will be 
small . 

There are at least two other ways to deal with the 
multidimensionality of achievement tests in a tailored 
testing application, but at this point neither way is 
practicable. One way would be to sort the test items into 
unidimensional subsets, and treat these subsets separately. 
However, thus far there are no satisfactory procedures for 
sorting items into unidimensional subsets when the items are 
dichotomously scored, which achievement test items typically 
are (Reckase, 1981). Even if sorting could be done, the 
problem of insufficient items in the pool would still be 
present. 

The other way of dealing with the multidimensionality 
problem is by using a multidimensional model. 
Unfortunately, no one has yet developed tailored testing 
procedures for a multidimensional model. Therefore, this 
study took the approach of using a unidimensional model with 
individual content areas. The content area used was the 
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math subtest of the ACT Assessment Program. Using these 
items, a pool of 40 items was constructed. Using this 40 
item pool, a comparison of the 1PL and 3PL models was 
conducted. The results of that comparison will now be 
discussed, beginning with the real data part of the study. 

Real Data Analyses 

Item Pool Calibration Probably the most significant 
result from the item calibrations was the finding that the 
3PL item parameter estimates yielded a test information 
curve that was negatively skewed and centered around a point 
on the positive end of the ability scale, while the 1PL item 
parameter estimates yielded a test information curve that 
was symmetric and centered around zero . From these results 
it would be expected that the 3PL tailored tests would tend 
to terminate prior to convergence for examinees with ability 
on the lower end of the scale. Such a tend€.ncy would not be 
expected for the 1PL tailored tests. 

Ability Estimates The most important finding from the 
analyses performed on the ability estimates obtained for the 
real data was that the 1PL model performed as well as the 
3PL model without requiring any additional items. The 
correlation between the 1PL and 3PL tailored testing ability 
estimates was fairly high (0.772), and the 1PL tailored test 
estimates were just as highly correlated with the paper and 
pencil test estimates as were the 3PL tailored test 
estimates. From these data it appears that there is no 
advantage to be gained from using the more complex (and 
expensive) 3PL model. 

Average Test Length For the real data tailored test 
simulations, the average test length for the 1PL and 3PL 
tests were about the same. This is as it should be, since 
the information cutoff values for the two procedures were 
selected to produce tests of equal length. 

Nonconvergence There were no cases of nonconvergence for 
the 1PL tailored test procedure. For the 3PL procedure 
there was a 4.9% nonconvergence rate. Of those cases where 
there was nonconvergence, 96% involved examinees at the low 
£nd of the ability range. This is consistent with the 
finding that the 3PL test information curve for the item 
pool was negatively skewed . Clearly nonconvergence is more 
of a problem in this case for the 3PL procedure than for the 
1PL procedure. 
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Simulation Data A naly ses 

Intern Pool Calibration What turned out to be one of the 
most important results of the item calibrations was that for 
the 1FL calibration LOGIST assigned to the items a common a- 
value which was higher than that assigned to the items using 
the real data. This resulted in higher test information for 
the 1PL model across the ability range. As a result of 
this, the information cutoff for the 1PL procedure was 
inappropriately low, which resulted in the tests being 
longer than expected. The test information curve for the 
3PL model was also somewhat higher than for the real data, 
except at the extremes. This would also be expected to 
increase the average test length of the 3PL tests, but not 
as much as for the IPm tests. The 3PL curve was negatively 
skewed, as was the case with the real data, which should 
have once again resulted in some none onverge nee cases at the 
low end of the ability scale. 

Average Test Length As was expected, the average test 
length increased for both procedures. The 3PL average test 
length increased by a little over one itenw while the 
average test length for the 1PL procedure increased by about 
five items . There 1 s no reason to assume that the quality 
of the 1PL ability estimates would have dramatically 
decreased had the 1PL tests been shortened by several items, 
although it would probably have been lower. 

N onconvergence For the simulation data the 3PL 
nonconvergence rate increased to 5.9%* while the 1PL 
procedure had a .3% nonconvergence rate. Once again, 
nonconvergence is clearly a more serious problem for the 3PL 
procedure than for the 1PL procedure. As was the case for 
the real data* the bulk of the nonconvergence cases for the 
3PL procedure (90%) were at tho low end of ability. This is" 
cons J stent with the results oL the test information curve 
analyses for the simulation data item pools. 

Summary and Conclusion s 

A study was conducted to compare the 1PL and 3PL models 
in tailored achievement testing application. Both real and 
simulation data were employed. For the real data, the 1PL 
procedure was found to yield ability estimates that 
correlaced with paper and pencil test estimates as highly as 
did the 3PL tailored test ability estimates. The 1PL tests 
were of about the same average length as were the 3PL tests. 
For the simulation data, an inappropriately low information 
cutoff was used for the 1PL procedure* and as a result of 
the 1PL tests were on the average four to five items longer 
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than the 3PL tests. The 1PL ability estimates were found to 
be significantly more highly correlated with paper and 
pencil test estimates than were the 3PL estimates. It was 
unclear what the results would liave been had the 1PL t^sts 
been terminated earlier. 

The 1PL model is a more appealing model than the 3 PL 
model, since it is simpler to work with, requires smaller 
sample sizes, and is overall much less expensive to use than 
the 3PL model- The results of this study indicate that for 
this type of high quality, small item pool, there is no 
justification for the added expense and complexity of the 
3PL model. For this application, the 1PL model was found to 
be the model of choice. 
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UisMn<;ton # D-1 20150 

1 Dr. Atfrct F. StioJ^, Director 

Training \n.ilysla 4 tivalmtlon Group 
D^pt . of th t Nivy 
Orlnnlo. FL 32H11 

I Dr. Rl^hirl Snren**n 

H ivy Personnel RSO Cont^r 
Sin Dlc^o. C\ 92152 

I Dr. Fr>*Vrtck Stcinh *tspr 
- 0PU5 
Navy Ann*** 
Arlington. VA 20770 



Navy 

1 Mr. Bn4 Synwson 

Hivy P^rsonnM ft*D *>nter 
Sin DU*o, C\ 02152 

1 Or, Fnnk Vlclno 

Nivy P.*raonn-*l R^O ^Mitor 
Sin Dii^to, ^ 92152 

t Or. £ Waf4 U *pjn in 

OffJn> of Miv-il R'H^ir-li {OH • VUS^P) 
HM North 'Vtinry Serine 
\ri tti«lim, V\ :>?!17 

I Dr. K'xriM W»Uzum 
M il Po^t (jri'tint* 1 S^hv* t 
0 *p irt*t» *n # , of Vlmlnlsmit I 

Sri^m-*** 
M-int-ivy, C\ 019'tO 

Coin 1; 

Sivy P^rsonn >l R^D C- # nt< # r 
I 'di. M\Hni F. WISK1FF 

swy perstmbu r*. n center 

S^N DIHCO, ^ 02152 

I Mr John II, UWf* 

Nivy P**rH*>nn*!l RV) Vntf*r 
Sin Otiirjo. CA 02152 

i Dr. Millar Utiifnck, Ut 
Nivy PtswhiM RM> Center 
Sin Dt.^o. C\ 92152 
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Mirlrv: Corps 



Army 



1 It * Wt I Usm fJrc *nun 

Glut-it ion Mv/tsor (BUI) 
fMurulnn Concur. HZftW 
'>> int trn , V\ 221 t'» 

1 Dtr-f tor, of fit— of Minpo^^r UtilUitin 
HI. ^irtn- Corps <MPi|) 
ROB, »M(f. 2«)T> 
0'i intko, V\ ^21 U 

I >l • 1 irt'-rtf , II* S, • 1 1 r i ii • i-'irp^ 
Co I * Ml* I -JO 
U K'lln'Lon , 11" J 1 11 I 

I Sp t I i I Assist mr for Mirlo- 
C'»rns v i >t t *»r** 
Co U> ID M 

Of f lr*- )f Nw il R ir'Mi 
81 I N t O itn'W St. 
\rl |orof) ( V\ 2*717 



I Technl Dl f ^r.tor 
U. S. \rny R-^nrch Institute for tht 
BHiwlnril .,rH ^ t % I ^ci^nr^s 
V)*)l Elifn'i'wr Av.-nu- 
Mttx-irvlrl i, V\ 72^11 

t ^Ir- Bik- # r 

\rmv R , s--irrti 1 n*i t i cn r 

M-k mlri i # v\ 2? m 

1 Dr. K*nr itoi 

\riw H-*s-'ir^h Inst I cut" 
V) M Kl s-MiMvr ftl v I. 
Al' x inlM i . V\ "»! n 1 

t l)r. H * itrh"» I. F »rr 
IK S. Army R^»ir<*1i lnsr|rut' # 
f »T*i E i H'»n*tiv * r (>nu * 
A^-x-nHrl i # VA 27 n \ 



I in, A.1-, SUFKOWf 
111, U.S. '-IVUMfi OO^lK 

w\siiiN"roN, in 2) 

1 Milnr Fr ink Yohmnin, U'^" 
11 MlqmrtTS, Urin- C->rps 
(Colo HIM-30) 
Wisttini'Cnn > D\ 2=) W t 



I Dr- Myrnn Fi*rhl 

U.S- Army R'»t-irrh Intti tut" for tit' 

I j i I ml !l- ,l n v I n r.i \ $c\\**\**f** 
VD1 El s a h 1 ihwt A*'f*nij • 
Al-»x.irvlrl i # V\ 

t Dr. HI itnn S. K*tz 

Trtinln^ Ttn'mlnl \r*' t 
II, S. Arrov ft-*<-*irch Ini*:t tut** 
V) 1 1 Et H«nhov * r Av-'no ■» 
M.»x-imlrl i> V\ 



I Dr. Ihrold F. o'N-M I . Jr. 

D|r-!rtnr, Tr lining R^s^irrh Lih 
Army ft^s<*irr*i ln*t(tut<' 
VV) I El i^nlioy »r Av< # nit • 
Al-TKinlrl t. V\ 2?,1H 

I Coi*mni>r, U.*>. Army H^vireu Institute 
fnr th * B-hwInrnl * Sorl.il Sol^nr.** 
ATTN: PBRt-BH (Or. .lulHh Orimnii) 
5 "HI KhonHrj»r Avniu # 
Al<*xin<1rli, V\ 20 111 

I JnS'*pS Psntk> t Ph.D- 
ATTH: PEHI-IC 
Anw R-'^Mrrt t n*t 1 tttt* # 
5'10l Gtn-nhiwT A^* # - 
Al'JxaiHrii. V\ 
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Army 

t Mr. Robert Rosa 

M>S> Araiy Rise-iTch Institute fnr th * 
Soci it -in I fl^bivinr it S<*ltfir'*s 
V)0l s*»nhitf^r Avi*nu * 
M?x.in-1rii. V\ 22m 

1 Or. Robert S.isnnr 

IK S\ Army R*'s^ir**h lnstltut* for lh" 
H *1i w lor-i I ml 1 il S-i«*uc *i 
V)OI Els >n*i*w**r Avc*au • 
Mmjc in Irl i , V\ 22 HI 

1 nr. lovt* Vih*l:ls 

\rny R*»Heir**h Instituo* for tV 
H »h 1 u 1 I 10H i H S** Ii*:i/-*h 
S )01 E I m*nhov»r Av(*nu * 
M »x indrl i. V\ 22U1 

I Or. Ill H.i Uinfi 

Army H'*» mtHi I ns r t 1 tut** 
S 1 M Eii-Mihiw * r A\**'. 
All-in Iri.i , V\ 22 Hi 

I Dr. Hob >rt UUli *r 

Army R*»d»irMi Institute 
VK)l El S'*o1nw ♦ r Avunn > 
\1<>xanlrli. V\ m\\ 



\ir Force 

I Mr Forc^ Himm R^siiiroos Lib 
AFHU/MPI) 

Brooks APR, TX 7R21S 

I T<*rlnfoil Dnoim?nts C«»nt"r 

\lr Fore* Him in R^sitir^s Libir.itorv 
Ui'AKH. 0:1 4V#H 

I Atr F*yrr*> Offlr-> nf Sci«nt1f1r 

R**s^ irdi 
ltf<* S-.lcnr" s 01 r**rtor it«% NL 
H )l 1 1 \\ r For^" Hn» 
U i^litn^ron. TK 2'U12 

I Atr Unlv;rstry l.lbriry 

AUL/LSB 74/U1 
- Mixvrrl t AFH, Ai. V)1 12 

I I)*" . Rirl A. M InUI 
H ), AFIIRL (\F*i*> 
ftronks \K*1, T< 7H2 IS 

1 Mr. Riy.Ai'i*! E. 0*irlstil 
AKiiRUMOE 

ftrooks AKH« TX 7S*>1S 

1 Dr * A 1 f r<»ti R. Fr^^l y 
VWR/M, 

BtI I AF<1. IK 20112 

1 Dr. Onjvleve Hi4H?H 
Pro^ran Mmn-r 
Life Sc1^nc**s 01 r^tor.if* 
AFOSH 

Hnltim \Fft. OS 20112 

I Dr. T. M. Lon^rlHt*'* 
AFIIRL'OTE 

WilliiDs AFB. KK HS22'# 

I Mr. Rin Intph P-irk 
AFHRL/MOy* 

Brnoks AFft. TX 7*21S 

1 Dr. Rc^or P *nnM i 

Atr Fnrco Himm R »&4nrci»s Libontory 
Lovry AKfl, CO 81210 

I Dr. Malcolm R<*e 
AFilRL/MP 

Brooks AFR. TX 7S21S 
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Mr Forct; 



D^pirttnon*: of D>ft!n*n 



I 3700 TCH(V/TrG*R 
2l.t TUlarl^o 
S*i:npir1 AFA, TX 7611 1 

I I.e. 0>L J imjG E. Uitson 

in usxfMpxh 

Tli** P**nt isf*on 
AV -'IPO 



12 D"f^w T rt chilc-*1 tnformtlon Ontir 

Al-jfti-vlrf i, V\ 22^14 
Utf»; TC 

1 Dr. Cr-ni^ I. Fields 

^ v in^H R«»virc*i P r o |<M:ts Aqpnrv 
140) iJLlsin Vv). 
VI tn*U<m. V\ 72109 

1 J<*rrv 1.***mu<* 
MO tf\*VY\ 
V tn: H'CI*"r-l* 
Fort SVrHin, H. 'i m> 

I Oltltiry Assl^ wt for mining in I 
P*r»ionnM T— Inolo^y 

')ffU*i» of th* MMf-r *Vrr*nirv **»f » *f ^nn 
for R**H"irch ^ E'i**ln A i»rln:» 
Ro.vn *ni?,0. TV 1>-n! i<jon 
Wiqiilotttoa. 0"* 2*H01 

1 Or . W iyn'< S-l tn hi 

Off to* of tV Assist inr 4 S"rr*t iry 
of n>f.*n**** (MH\ * 1.) 
2TW) r*i • P*nt u*on 
vJi^Mmjr o»i, 20V) \ 

\ M \ lor Jick Tnorp » 

n\«p\ 

1400 Lson Blvl. 
\rUi>.{r»a f V\ 22209 ?/ 
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Civilian 4(V»iKtG$ 

1 Or. Susm Chlpum 

L^mln* -u\i F)£vn1opn»nt 
Nittoml tn«tttuM of Eiucittoa 
1200 t<Jth Strcsit NN 
Wi*hln*ton. 202<>3 

1 r>r. V'iro W, Urry 
Porsono *1 RSI) '>nter 
Office of P*'rsot»n.»t Mio i^n-nr 
190) F. Stri 1 *t 
Jishtmjton. IT. 2^**1 S 

t Hr. Tlrvn i<» A* W»rm 

IK S, 0>ist G*tiM Institute 
P, 0. Sub*tt UtoT (ft 
Oct i»ivs i CI tv, OK 7 T 1 -S 1 * 

1 Hr. JnaitVi L. Yonm;. Dtr*H*tor 
tii-no ry f* Goffntttti- Pro.vsa.n 
Nittomt Scti'o^,* F'ui'H ittoo 



Private Sector 

I Or, Jn-ac* M^fnt 

Untv^rstcy of FlnrHi 
GilivsvMtl". Ft 12^ 

t Or, Erllnt* ft, AiH-rs^n 
O^n-irtm^nt of ^tictsttcs 
StiH r v 1 !" 
14S*i Cop'ohi^"! 

I t Psvrhilo?, \c\\ R's^irrti Hilt 
Wl-l-'*/* Utn: Uh'irlin 
NirthS >unv* II ***** * 
Turn t \cr 2ft'H 
VJ'iHUUA 

I f)r. Unr ft >\ it 

E1ii<* i^lon it T^thm Vrvlc 
Prim*" ton. HI flViSi) 

\ Hr. M'onrli i IHivnb inn 
•;/ln->l of K in' it to:> 
T-l Avtv Uilv-rtltv 
T»'l A*1v. Ri-nir Aviv W97S 
1 *r i^l 

1 Or. ft, Dirrcll tWi'k 
I) *p irtrti**iit of £ hi'- 1' 1 (in 
Unlvrrsltv Oilriqo 
CM^Ko, 11 WI7 

1 Or , Robert ftr-*nn in 

Vn^rtr'in Colt^** T<**tttn*t Pr#H*rvm 

P> 0. ftox IftH 

Inwi Ctty. I\ WVt \ 

1 Or. Ernest R. Cvlott" 
VI 7 StoVly 

UnW'*r*1tv n f Temviw 
Kioxvlttr*. Ttf 1711ft 

t Or, John B. Cirrol 1 
409 Elliott M. 
Chipi't Mill. « 27^t4 

t Or. Nirmin Ctlff 
0>pt . of P*v<**nto7,v 
Univ. of So. CittMrnla 
University P»rk 
t^s Amv>t*><*. ini; 
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Prtv.itt Succor 



Prtv-)C<* Succor 



1 Dr. Ibns Cronbu* 

Education R'».S'j-irch Oint-T 
University of Liv^n 
It > % rh \a v i I -i in 2 
21 T* E*J Uyl *n 

1 Dr. t) »tcpr id il DtvpJ 
Syr »**'ISi* Unl v<*r* I Ly 
I) »p irtn»nt of Pf*yclolo:*y 
Syr i< <is - . N v * U2l'l 

I Dr. Frttz t)r i^j'w 

l)*pircm-n* of PiYr-»i >1ohy 
tin 1 v«*rnl t y of 11 1 1noH 
f>') I E. n in I Sc . 
Ch inpil<*n ( 1 1. 61 

1 Or. S-is in Eiih >rcson 
PSYIIVILKSY l)?-P\HT|^r 

UN 1 vr.t*? tty or <vi5v; . 

I ( iwr-*ni"*, KS ^> , )'*-> 

1 ERIC FnMUiy-ArqMi*ldon<» 

H*Chi*<H i, MO 20 )H 

I Or, H *ii 1 -m l n A. F.il rb ink. Jr. 
M :F mn-Cr-iy ^ Assoc! ic^s, Inr, 
Vi2S Ci1 i.ntinn 
S i I Co 2i<> 

Sin \ntonlo. TX 7<122ft 

I Or. I'omrl FMdt 

Ltndqnlsc C-'nCiT for HMqurmint 
Uol v^rnl Cy .of I o j i 
1o«n Ctcy, tA <>2242 

1 Or * ch ir-l L. Fur^nson 
Th" An-Tlcnn Coll'**v* IVscln*, Pro^nm 
P,0. I\m< 168 
towi City, lA *>22'*0 

1 Univ. Prof. Dr. C*rtnr*l Flanhsr 
Lt *:bti;in4V* V J 
A 1010 Vlennt 
AUSTRIA 

1 Professor Dintld Flcz^nl 1 
UntvcrsLcy of H "j Rn^tnnl 
\rmldile. H->v Sonch Wiles 2*51 
AUSTRALIA 



1 1r, D »xcc r F1 ^c cli^r 
WICVT R^inrch Institute 
1 37^ S. stu* SC. 
Orun. ur 22 m 

1 Or. J into* CI f for<1 
il'il v*rsl cy of *i iss ich is-ccs 
School of Edii'-ition 
A-nVra^ , M\ 0)002 

I Or i Rob-r i Ct is^r 

1.->rntn-* H-virrh S D'voiopv^n' C*n*-**r 
;Hlvi«rslcy of Pittsburgh 
i"l 19 )'n ir.i Scr-*-c 

rrrrsnti-uri. p\ rPf>o 

1 Or. H^rt Cre;*n 

lohus Hopkins Unlv^rntcv 
D**p trc<n<*m of Psyrholo^v 
C«nr1*'s *> Y*V\ Street 
ttilrlmtr-, UO ^ 1 2 1 H 

I Or. Ron II mblccoo 
Srhool of Eduction 
Unlv**rslcy *if H isvichns *c ca 
A-a^rnt. *l\ 010*12 

I Or * D 1 \ wya >h rnlsHi 
Mn1v^r3lty of UltonH 
2'*2h c ,-1ti^ ic ioo 
Urbin i. it M«H 

1 Dr. I'm I Hors* 
677 C Scr^c. fflV* 
Chnl a Vise*. CA 90010 

I Dr. Lioyl Humphreys 
D'Pircm.'nc of Piv^lnlottv 
University of Illinois 
601 Rise Oinld Scrc<*c 
Chmpiis;n. IL 6l«20 

1 0^. Jack II in^r 
2122 Coolld^r. Sc. 
Linatn^. Ml W)h 

1 Or. U'iyo*i "luvnli 

Colluf/ of E-Itjcndon 
Uolv^rslcy nf SoiiCh Ciroltm 
Colunbln ■ SC 2920^ 
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Private Sector 



Pr t v-ito tytctor 



1 Dr. H . Jones 

Adv weed Statistical Tactnolortt^s 

Corpor.it Ion 

10 Tr lf.iltjir Court 

Liur^nc*vltln , NI OHUV 

1 Prof-ssor John \ t K» its 
D:»j> ir'.ni^nt of Psy Oology 
Tlv lln I v- r s 1 1 y of N^wistl" 
:J.<>,W. 2\n 

L Dr. Willi »i» Ko.:h 
Ual vt*r*#l ty of T'xn-Aqsitn 
H * m»»r^ >nr .ml Gv iluitton C-*nr*«r 
Austin, TX 

I Or. Al-in t,ff?<(ot<1 
t.iinil n^ R*n ^nt.;r 
Unlvralty of PI t tnh-iri*h 
m* OMI-ir i St r^-t 
PiltHh-.iri'h, P\ I S2-'iO 

I Dr. Mtch i«tI L^vln-* 

0 *t>i»rtm^nt of E'ltKit Io.tiI P<jyclv» I o,*y 
210 £<tu"nlon B11r,, 
Uol viTt; 1 ty «>f 111 InoU 
CVimpitirn, IL 41 VH 

I Or. Ch ir I -*n L'jwl 9 

F-icul tf?t t So -Mil'* W ♦tunned ipp^n 
Ht ]k4unt vrs I tet t Groninvn 
fV|-to ItotcrlnP, -3t r.i it 21 
9712CC Groninq.'n 
N 'th^rt/in-ls 

1 Dr. Robert Lino 

Collide of E'lucotlon 
Unlvrrslty of Illinois! 
'Jrhini, it. 61 HO) 

I Mr. pl»l 1 1 1 p Liv |ni{ston 

Sy^ttftt* ml \poM**4 Sci^nc^ Corp-iruio 
6*U 1 K^nilworth \vf*nu'> 
Rtvi»r.UI<»» MO 2ov#o 

\ Or. Rohert Lock-inn 

C »nt 'jr for Nivo I An il ysis 
200 North B^iurctjnrl St. 
Mcxinart.i. VA 22111 



1 Or. Frederic M. LoH 

E-im.ntlonil Tdsttnq Wvlc<* 
Prlnr.iton, N1 fV*S'*l 

I Dr> .lim^s L'nisi^n 

l)"!>irt*n'»nt of Psyliolo^y 
University if U>st<»rn Vintritli 
S -41 in-l-s W.A. ^9 
MJSTRU.I \ 

I Ur. Girv Miroo 

?A\tr M I it TVsUnt* S-rvl'*'* 
Princeton, Hi 

I nr. S*otr. 'Iixwl I 

D *p irtm «nt of P'iyohnlo:*y 
U:i1vi>r*ity M Nitre Dimo 
Nntn* Ovm*. IN Wtt 

I Dr. S m-v»| T. Miyo 

Liyol i University nf CM^i^i 
<*2'l Nortli Ml chimin Av*>nn > 
Clilr.140, tl. *1S1l 

I Mr. Ron *rt M^KinMy 

Aii'*rir»n Call^n" T^stlnq Pro^rims 

P.O. Box 16H 

lmn City, l\ S2241 

1 Or. BirMn M '-ins 

'hinin R -sources U'^^irrh Orq ini r it ion 
V) 1 } North Wiahlnnroi 
M^xnnlri-n. VA 2211 A 

I Dr. Rohert Mlslcvy 
711 Til innts St r^t 
f>n-vn, IL *01 1'* 

1 Dr. Al lf*n Mnoro 

Behavior*! Tnohnolo^y Libor/itorlrs 
IfHS Elnm Ave., Fourr.h Floor 
HHondo B^ich. C\ 9027 7 

I Dr, W. Aim N1 c^w m.lrr 
University of O'-tlohoin 
D*p»rtm» # nt of P<jy**ho1 oc,y 
o'tlaho-ni City. OK 710S9 

I Dr. M-^lvin R. Sovlck 

Llnlqulat C-nti*r for M*"i4urji*nt 
University of to-va 
towi City. tA 
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Private Sector 



Private Sector 



t Dr. Jimft Olson 
WtOVT, Inc. 

I ft? S South St ltd Stro:!t 

f>rt!i», ur a'*os; 

I :j lyne M. Pat tcno: 

An 1 !'* t c in Counr 11 on tHu** it 1 oi 
GEI) Testing S»»rvlc<*» S*ilt" 2') 
Oi^ Oipont Olrlo, 
riisliin-ftni. 0" 20) I S 

I Dr. J \m i( i A- 1* ml **>n 

Port Tin I St it" University 
P.O. 

Portl iun\ OR 
I Or. Mirk 1)* R >ck is t 

v:r 

P> 0. Ho* If*H 

town Cl'y, tA 52V* I 

I Dr. Thorn* K»y«iolJs 

University of Te.xas-0 tl 1 ati 
Mirkulni; I) 'pi rtn^nr. 
P. 0. Bo* S<H 
KlchinWm, TX 7Vm 

1 Dr. Liwreni*** K-Hnsr 
4^1 gU Nvenu > 
Tnkom Pirk> Ml) 20^12 

I Or. J. Rym 

Department of R*luz;i t ion 
University of South Ctrollni 
Columbia. SC 2920* 



I Lowell Schorr 

Psyrhnloftlcnl & *)uint 1 1 i 1 1 v<* 

Foun*li t ions 
ColIe^T of EJucitton 
Uni v^rsl ty of to>*ri 
low i City. U 5?2'*2 

i dr. mif.nr J. skih^l 

INSTRUCTIONAL TE^mL'ttY CtO!IP 

i *o i. v\sin>nr<T4 ^r. 

\UvX\Niym. V\ 22H'* 

i Dr > Kifciu Shin *i ism 
University if Toh*»k»i 
D'i> irtrt^nt, *f Etucittonil )NyHiolo**v 
Kiwaueht , S**n1 it OHO 
-HP VI 

I Dr. E-lvtn Shirk >v 

0 *pirtm**nt *f P*yrhn 1 o^y 
University of 0»ntnl FlorHa 
Orlanln, FL l^H t ^ 

1 Dr. tfllltm SUs 

Oat<*r fnr 4w*i1 Viilytls 
20'> North H-nure^ i rd Street 
Ate*in*ri i, V\ 22111 

1 Dr. It. UiUir.' Slnitko 
Pro<?r.in Dl rer tor 

Manpnvr K" 1 *;* irrh in t A.lvlsory Services 
S-nttliioninn fnstltorlnn 
801 «nrth Pitt Str^r 
Al^ann-lrln. V\ 22H4 



i prof. fi1mik0 sams j i m\ 
d*s:t. of p^y^hi/isy 
university of tennessee 

KNoXVlLLE, TN 17<M6 

t Frank L- Schutilt 

D»pirtn*nt of Psyrh->1o^y 

O^orQ-* Wishlnqtnu University 
W ishln^ton, DC 20052 

I Dr. Witter SchnuhW 
Pnycholo^y D *p irtn^nt 
ftOl fi, DintM 
C'nmpU^n, tL MH20 



1 Dr. Rnb?rt Sttrrnb^rf* 
l)<*pt. of PsfrhDlcxry 
Yale University 
Bd* IU. Y tic* Stitlon 
N^w 11 wen. CT 06520 

t Dr, P*t*r Stoloff 

Center fnr flw.il Anly*ls 
20*> Vorth 4 Miirtig.ini Strn-*t 
Alfxnnlrl.i. VA 22HI 

t Dr. Ml 1 1 in* Stout 

Ua lv:rslty of til inol* 
D'Oittii^nr of Mithroitlcs 
llrbini. It 6lft01 
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Private Snetor 

1 Dr. ltirltvirin Sur&ln.ithm 
tibontory of P*yr,ho:n *trlc .ml 
Rv*iiu it ton Ros**irch 
S-nool of Eiocatloi 
University of H ist-tchns :t tn 
VnVrst . M\ 01 D11 

I Dr. Klkuitt T-itsuoki 

Conp.itur B id-!-! F-liic-i' ton R'S'Mrch l,ib 
2 S3 Ra^tn^^rlat* R*»S'» irch t.ibor uorv 
llfbmi. It. MBit 

I Dr. *twrtO' Tittuoki 
22') *tu-itloii lid* 
I J 10 S. SUth St. 
ChinpH«n. tt MH20 

I Dr. Dwld TMss^n 

I) *p irtm^nt of P*yHio lo^y 
Unl vcrn t cy of K ins is 
tiur*'are t Vj'H'i 

t Dr. R'»bort Tsutukiai 
D-partm^at of St.ittstlcs 
University of Ml snout t 
Cnlunbti, HO f>520l 

I Or. J. Uhl in**r 

Uht mur Consultants 
42 r »ti Bnnwtt.i Drtvo 
Sncino. C\ OHW 

1 Dr. V. K. R. UppulurJ 
Unton OirbMn Corpor-itloo 
N'isleir Dtvtston 
P. 0. B3K Y 
Oik Rld<t* t 17810 

1 Dr t D.ivld V-il« 

A*i*cs**n j| nt Systems Oorpnr it ton 
2211 University Av^ou* 
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